Terminal device, program, method, and system

ABSTRACT

A terminal device includes a pseudo input interface configured to make a user to perceive a pseudo operation, an external communication interface configured to receive intention information indicating an intention of the user from a sensor device, the sensor device being wiredly or wirelessly connected to the external communication interface, the sensor device detecting the intention of the user who intends to perform an intended operation with respect to the pseudo input interface, a memory configured to store computer readable instructions, and a processor configured to execute the computer readable instructions so as to identify the intended operation by the user based on the intention information received by the external communication interface, regardless of an actual operation perceived by the user with respect to the pseudo input interface, and perform a processing corresponding to the identified intended operation.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation application of InternationalApplication No. PCT/JP2019/014734, filed on Apr. 3, 2019, which isexpressly incorporated herein by reference in its entirety.

BACKGROUND 1. Technical Field

The present disclosure relates to a terminal device, a program, amethod, and a system which use a detected intention of the user.

2. Related Art

When a human receives external stimulus, the human perceives thestimulus with a sensory receptor, judges the stimulus, designs what todo, and finally contracts peripheral muscles to take action. Generally,it is known that it takes 200 milliseconds to take the action afterreception of the stimulus even in a case where there is no load (HitoshiKawamura et al. “A Research Electromyographic of Reaction Time (No. 1)”,Bulletin of Aichi Institute of Technology No. 7, Mar. 30, 1972, p33-43).

A terminal device, such as a smartphone, is operated under the conditionthat judgement is required for various stimuli received from variousfactors other than the terminal device. Therefore, it is readilyconceivable that it takes 200 milliseconds or more to input aninstruction after receiving a stimulus from a display.

Furthermore, in an input interface, it is necessary to perform tasks,such as moving, positioning, and touching (click) of fingers or a mouse,even after contracting the muscles to start the action. Therefore, it isknown that it takes 200 milliseconds to 1,200 milliseconds to completeall the tasks (Masanobu Komazaki, “A Study of Human Pointing Features onTouch-Screens”, Doctor Thesis, The University of Electro-Communications,September 2008).

Meanwhile, it has been known that detection and measurement of a brainwave of a user enable to measure an intention of the user on the basisof the measured brain wave to operate various devices. For example, inJP 2007-202882 A, it is stated that a user's attention area andattention degree are measured on the basis of a brain wave of the userto control a movement direction and speed of an electric wheelchair.

SUMMARY

On the basis of the technology as described above, the presentdisclosure provides a terminal device, a program, a method, and a systemaccording to various embodiments which perform an operation input morereadily than a user actually operates an input interface by usinginformation on an intention of the user.

According to an aspect of the present disclosure, provided is “aterminal device comprising: a pseudo input interface configured toreceive a pseudo operation by a user; an external communicationinterface configured to receive, from a sensor device wiredly orwirelessly connected to detect an intention of the user who intends toperform an operation on the pseudo input interface, intentioninformation indicating the intention of the user detected by the sensordevice; a memory configured to store a computer readable instructions;and a processor configured to execute the computer readable instructionsso as to: identify an operation by the user based on the intentioninformation received by the external communication interface, regardlessof an actual operation performed on the pseudo input interface by theuser; and perform processing corresponding to the identified operation”.

According to an aspect of the present disclosure, provided is “acomputer program product embodying computer readable instructions storedon a non-transitory computer-readable storage medium for causing acomputer including: a pseudo input interface configured to receive apseudo operation by a user; an external communication interfaceconfigured to receive, from a sensor device wiredly or wirelesslyconnected to detect an intention of the user who intends to perform anoperation on the pseudo input interface, intention informationindicating the intention of the user detected by the sensor device; anda memory, the computer configured to perform the steps of: identifyingan operation by the user based on the intention information received bythe external communication interface, regardless of an actual operationperformed on the pseudo input interface by the user; and performingprocessing corresponding to the identified operation”.

According to an aspect of the present disclosure, provided is “a methodfor causing a processor in a computer to execute a process, the computerincluding: a pseudo input interface configured to receive a pseudooperation by a user; an external communication interface configured toreceive, from a sensor device wiredly or wirelessly connected to detectan intention of the user who intends to perform an operation on thepseudo input interface, intention information indicating the intentionof the user detected by the sensor device; and a memory configured tostore the computer readable instructions, the method comprisingexecuting the computer readable instructions on the processor the stepsof: identifying an operation by the user based on the intentioninformation received by the external communication interface, regardlessof an actual operation performed on the pseudo input interface by theuser; and performing processing corresponding to the identifiedoperation.

According to an aspect of the present disclosure, provided is “a systemincluding the above terminal device; and a sensor device wiredly orwirelessly connected to the terminal device to detect an intention of auser performing one or a plurality of operation inputs to an inputinterface of the terminal device”.

According to an aspect of the present disclosure, provided is “a systemincluding the terminal device described above, and a sensor devicewiredly or wirelessly connected to the terminal device to detect anintention of the user performing one or a plurality of operation inputsto an input interface of the terminal device”.

According to an aspect of the present disclosure, provided is “aterminal device comprising: a pseudo input interface configured toreceive a pseudo operation by a user; an external communicationinterface wiredly or wirelessly connected to receive a signal indicatingan intention of the user performing an operation on the pseudo inputinterface; a memory configured to store computer readable instructionsand a correspondence relationship between the signal and the intentionof the user; and a processor configured to execute the computer readableinstructions so as to: generate intention information on the intentionof the user based on the received signal and the correspondencerelationship stored in the memory, regardless of an actual operationperformed on the pseudo input interface by the user, identify anoperation by the user from the generated intention information, andexecute processing corresponding to the identified operation”.

According to various embodiments of the present disclosure, a terminaldevice, a program, a method, and a system are provided to perform anoperation input more readily than a user actually operates an inputinterface by using information on an intention of the user.

It should be noted that the above effects are by way of example only forease of description and the present disclosure is not limited to thedescription. In addition to or in place of the above effects, any effectdescribed in the present disclosure or apparent for those skilled in theart can be also provided.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram conceptually illustrating processing in a systemaccording to various embodiments of the present disclosure.

FIG. 2 is a diagram conceptually illustrating processing in a systemaccording to a first embodiment of the present disclosure.

FIG. 3 is a block diagram illustrating an example of a configuration ofa system 1 according to the first embodiment of the present disclosure.

FIG. 4A is a table conceptually illustrating an operation identificationtable stored in a memory 212 of a brain wave sensor device 200 accordingto the first embodiment of the present disclosure. FIG. 4B is a tableconceptually illustrating a processing identification table stored in amemory 113 of a terminal device 100 according to the first embodiment ofthe present disclosure.

FIG. 5 is a diagram illustrating a processing sequence performed betweenthe terminal device 100, the brain wave sensor device 200, and a serverdevice 300 according to the first embodiment of the present disclosure.

FIG. 6 is a flowchart illustrating a process performed in the terminaldevice 100 according to the first embodiment of the present disclosure.

FIG. 7 is a diagram conceptually illustrating processing in a systemaccording to a second embodiment of the present disclosure.

FIG. 8A is a table conceptually illustrating an operation identificationtable stored in a memory 212 of a brain wave sensor device 200 accordingto the second embodiment of the present disclosure. FIG. 8B is a tableconceptually illustrating a processing identification table stored in amemory 113 of a terminal device 100 according to the second embodimentof the present disclosure.

FIG. 9 is a diagram conceptually illustrating processing in a systemaccording to a third embodiment of the present disclosure.

FIG. 10A is a table conceptually illustrating an operationidentification table stored in a memory 212 of a brain wave sensordevice 200 according to the third embodiment of the present disclosure.FIG. 10B is a table conceptually illustrating a processingidentification table stored in a memory 113 of a terminal device 100according to the third embodiment of the present disclosure.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

Various embodiments of the present disclosure will be described withreference to the accompanying drawings. In addition, the samereferential marks are given to common components in the drawings.

Overview of System Relating to Present Disclosure

FIG. 1 is a diagram conceptually illustrating processing in a systemaccording to various embodiments of the present disclosure. According toFIG. 1, the system is configured to detect an intention of a user 11 toperform an operation input on the basis of a brain wave detected by abrain wave sensor device 200 mounted to the user 11 and performprocessing corresponding to the operation which the user 11 intends toinput, according to a result of the detection.

Here, in general, in a terminal device, such as a smartphone, an inputinterface with which a user operates the terminal device is a veryimportant component. For example, only detection of an intention of theuser on the basis of his/her brain wave and operation of the terminaldevice according to the detected intention are required to obtain auser's desired result, such as transition to a screen which the userdesires to view. However, in that case, since the user performs nooperation on the terminal device, it is difficult for the user to obtainthe feeling of operating the terminal device, and a poor operationalfeeling is provided. In order to obtain a satisfactory operationalfeeling, it is important for the user to perceive that the userhim/herself operates, for example, “pushes” or “touches”, the inputinterface of the terminal device.

Therefore, in a terminal device of the present disclosure, an intentionof a user is detected on the basis of his/her brain wave, and processingis performed according to the detected intention. Meanwhile, theterminal device includes a pseudo input interface to cause the useroperating the terminal device to perceive operation, such as “pressing”or “touching”, on the terminal device.

In the system illustrated as an example in FIG. 1, a display 111 of aterminal device 100 is caused to display icons 21 and 22 thereon. Theicons 21 and 22 are displayed to cause the user to perceive in a pseudomanner that processing assigned to each of the icons will be performedwhen the user touches the icons 21 and 22 with a pointer 12. Therefore,even if the user actually touches the icons 21 and 22, processing is notalways performed with the touching as a trigger.

On the other hand, when the brain wave sensor device 200 mounted to theuser 11 detects an intention of the user 11 to operate any of the icons21 and 22 displayed on the display 111, information on the intention istransmitted to the terminal device 100. When the terminal device 100receives the information, the terminal device 100 performs processingassociated with any of the icons 21 and 22. In other words, theprocessing associated with the icon 21 or 22 is performed on the basisof the information on the detected intention, even though it is notdetected that the user 11 actually touches the icon with the pointer 12.

In the present disclosure, the “pseudo input interface” represents aninput interface which is not treated as a trigger for performance ofprocessing, even though an operation, such as touching or pressing, isperformed. Thus, in FIG. 1, the icons displayed on the display 111 aregiven as an example of the “pseudo input interface”. However, the“pseudo input interface” is not limited to the icons and may be anotherdisplayed object displayed on the display 111, the display 111 itself,or a hardware key fixedly provided on a housing of the terminal device100.

Moreover, it is not necessary to always treat the “pseudo inputinterface” as the trigger even when an operation, such as touching orpressing, is performed. In other words, the pseudo input interface isdesirably configured not to be treated as a trigger only when a userintention detection function is on, according to a mode of the terminaldevice 100, an activated function/application, a selection by the user,or the like. Therefore, in the other case, the pseudo input interfacemay function as the input interface serving as the trigger to performpredetermined processing when an operation is received.

FIRST EMBODIMENT

1. Configuration of System 1 According to First Embodiment of PresentDisclosure

FIG. 2 is a diagram conceptually illustrating processing in a systemaccording to a first embodiment of the present disclosure. According toFIG. 2, when an intention of a user 11 to touch a display 111functioning as a pseudo input interface or press a hardware key 116 isdetected from a brain wave by using a brain wave sensor device 200mounted to the user 11, communication with a server device 300 isstarted, and necessary information is transmitted and received. At thistime, communication with the server device 300 is performed on the basisof the detected intention of the user 11, requiring no actual detectionof touching the display 111 or pressing a hardware key 116 with apointer 12 of the user 11. In other words, regardless of whether theuser touches the display 111 or presses the hardware key 116, processingcorresponding to the detected intention is performed.

FIG. 3 is a block diagram illustrating an example of a configuration ofthe system 1 according to the first embodiment of the presentdisclosure. Referring to FIG. 3, the system 1 includes a terminal device100 and the brain wave sensor device 200 communicably connected to theterminal device 100 wiredly or wirelessly. The terminal device 100 andthe brain wave sensor device 200 do not need to have all of thecomponents illustrated in FIG. 3, some of the components may be omitted,or another component may be added.

An example of the terminal device 100 includes a portable terminaldevice, such as a smartphone, capable of wireless communication.However, in addition, the system 1 according to the present embodimentcan be preferably applied to a device, such as a handheld game console,feature phone, personal digital assistant, PDA, laptop personalcomputer, desktop personal computer, stationary game console, musicplayer, printer such as multifunctional peripherals or laser printer,controller for a machine tool or manufacturing machine, car navigationsystem or vehicle equipment, as long as the device is configured toinput and output information to and from another device or module andreceive an operation input from the user.

According to FIG. 3, the terminal device 100 includes the display 111, aprocessor 112, a memory 113, an input interface 114 including a touchsensor 115 and the hardware key 116, and an external communicationinterface 117 including a wireless communication processing circuit 118connected to an antenna and an I/O port 119. Then, these components areelectrically connected to each other via a control line and a data line.

The display 111 functions as a display unit to read image informationstored in the memory 113 for various display, in response to aninstruction from the processor 112. Specifically, for example, in theexample of FIG. 2, immediately after a game application is started, thedisplay 111 displays a title screen for receiving a start operation fromthe user on a screen. Furthermore, although not particularlyillustrated, when an operation input to the touch sensor 115 or thehardware key 116 is received on the title screen, a start screen of thegame application is displayed. Moreover, these displays cause the user11 to perceive the possibility of operation on the display 111, and thedisplay 111 itself functions as the pseudo input interface. For example,the display 111 is constituted by a liquid crystal display.

The processor 112 includes a CPU (microcomputer) and functions as acontrol unit to control other components connected, on the basis ofvarious programs stored in the memory 113. The processor 112 executes aninstruction, that is, a program for a game application according to thepresent embodiment or a program for executing an OS, stored in thememory 113. Specifically, the processor 112 identifies an operation bythe user from intention information indicating an intention of the userto “perform an operation input” received from the brain wave sensordevice 200 and executes processing corresponding to the identifiedoperation. Furthermore, in some cases, when an operation input performedby the user is received by the input interface 114, the processor 112may perform processing corresponding to operation information identifiedon the basis of the operation input. The processor 112 may beconstituted by a single CPU but may be constituted by a plurality ofCPUs. Furthermore, another type of processor, such as a GPU, may becombined as appropriate.

The memory 113 includes a RAM, a ROM, and a non-volatile memory (in somecases, an HDD) and functions as a storage unit. The ROM stores, as aprogram, an instruction for executing the application or the OSaccording to the present embodiment. The RAM is a memory used to writeand read data while a program stored in the ROM is being executed by theprocessor 112. The non-volatile memory is a memory to/from which data iswritten/read upon execution of the program, and the data written to thenon-volatile memory is stored even after the execution of the program iscompleted. Furthermore, the memory 113 stores a processingidentification table (FIG. 4B) in which information about an operationdetected by the brain wave sensor device 200 is associated with thecontent of processing performed on the basis of the operation.

The input interface 114 includes the touch sensor 115 and the hardwarekey 116 and functions as an input unit to receive various operationinputs from the user. The touch sensor 115 is used to receive variousoperation inputs from the user, such as an operation of an icondisplayed on the display 111 and an input of a character string by theuser. The touch sensor 115 is disposed so as to cover the display 111and outputs information about proximity coordinates or contactcoordinates of the pointer 12 (user's finger, a stylus, or the like),corresponding to image data displayed on the display 111. The touchsensor of known type, such as a resistive type, a capacitive type, or asurface acoustic wave type, may be employed. In the present embodiment,the touch sensor 115 of capacitive type is preferably used to detect theproximity of the pointer 12 of the user. The hardware key 116 uses awell-known hardware key.

In the present embodiment, the pseudo input interface is provided toperform processing on the basis of a brain wave detected by the brainwave sensor device 200. Therefore, the input interface 114 is notnecessarily provided. The input interface 114 may be provided to be usedalong with processing based on the brain wave sensor device 200, orprocessing based on the input interface 114 may be performed only whenno processing based on the brain wave sensor device 200 is to beperformed.

Furthermore, in the present embodiment, the display 111 is caused tofunction as the pseudo input interface. However, the input interface 114may also be caused to function as the pseudo input interface withoutusing the received operation as the trigger for processing.

The external communication interface 117 includes the wirelesscommunication processing circuit 118, the antenna connected to thewireless communication processing circuit, and the I/O port 119 andfunctions as a communication unit. The external communication interface117 transmits and receives programs necessary for execution of variousapplications, user information, drawing information, and the like to andfrom the server device 300 installed remotely and connected via anetwork. In particular, in the present embodiment, the information onthe intention of the user who performs an operation input is receivedfrom the brain wave sensor device 200.

The wireless communication processing circuit 118 and the antenna areoperated on the basis of a wide band wireless communication system, suchas wideband-code division multiple access (W-CDMA) system or a long termevolution (LTE) system but may be operated on the basis of a wirelessLAN, such as IEEE 802.11, or a narrow band wireless communicationsystem, such as Bluetooth (registered trademark).

The I/O port 119 is connected to an I/O port of an external devicewiredly connected. The I/O port 119 can adopt a desired connection, suchas a serial port, parallel port, or USB.

In the present embodiment, various information including the intentioninformation are transmitted and received to and from the brain wavesensor device 200, but this transmission and reception may be performedby any of wireless communication via the wireless communicationprocessing circuit 118 and wired communication via the I/O port 119.When the terminal device 100 according to the present embodiment isused, it takes 200 milliseconds or more to perceive a stimulus by asensory receptor (perceive information displayed on the display 111),make a decision, determine an operation input, and finally contractperipheral muscles to take action (operation input to the touch sensor115). Therefore, in consideration of the time, a communication systemhaving a higher communication rate is preferably employed forcommunication with the brain wave sensor device 200, but thecommunication rate above a certain level is preferably required, and thecommunication system is selected appropriately from the viewpoint ofcommunication stability and communication rate.

Furthermore, according to FIG. 3, the brain wave sensor device 200includes a brain wave sensor 211 including one or a plurality ofelectrodes El to En, a memory 212, a processor 213, and an externalcommunication interface 214. Then, these components are electricallyconnected to each other via a control line and a data line.

The brain wave sensor 211 includes one or a plurality of electrodes Elto En. It is known that, in the brain, cerebral cortex generallycontrols the functions of perception, voluntary movement, thinking,guessing, and the like, and areas of the cerebral cortex play differentroles depending on the areas. For example, in order to perform anoperation input to the terminal device 100 with a finger, the functionsof the orbital frontal cortex, motor association cortex, and primarymotor cortex are required. Thus, the electrodes El to En are arranged atpositions on the scalp corresponding to these active areas. Thearrangement and the number of the electrodes El to En may be determinedin advance by sampling the brain waves of the user by using the brainwave sensor device 200 and appropriately selecting an electrode fromwhich the strongest peak monitored upon operation to the input interface114 is detected. An analog signal output from each of the electrodes Elto En is appropriately converted into a digital signal and used forsubsequent processing.

The memory 212 includes a RAM, a ROM, a non-volatile memory, and thelike and functions as a storage unit. The ROM stores, as a program, aninstruction for controlling the brain wave sensor 211 according to thepresent embodiment or transmitting or receiving information to or fromthe terminal device 100. The RAM is a memory used to write and read datawhile a program stored in the ROM is being executed by the processor213. The non-volatile memory is a memory to/from which data iswritten/read upon execution of the program, and the data written to thenon-volatile memory is stored even after the execution of the program iscompleted. In the present embodiment, the memory 212 stores an operationidentification table (FIG. 4A) in which a feature point of a brain wavedetected when the user intends to operate the pseudo input interface isassociated with the content of processing performed when the featurepoint is detected.

The processor 213 includes a CPU (microcomputer) and functions as acontrol unit to control another component connected, on the basis ofvarious programs stored in the memory 212. The processor 213 executes aprogram for carrying out an instruction stored in the memory 212.Specifically, the processor 213 refers to a brain wave signal outputfrom the brain wave sensor 211 and the operation identification table,determines a matching point with a feature point of a brain wave signalsampled in advance, and determines a content of operation which the userintends to input.

The external communication interface 214 functions as a communicationunit configured to be used to transmit and receive information to andfrom the terminal device 100. Accordingly, the external communicationinterface 214 appropriately includes a wireless communication processingcircuit 118, an antenna connected to the wireless communicationprocessing circuit, and an I/O port 119, corresponding to the externalcommunication interface 117 of the terminal device 100.

2. Information Stored in Memory 212

FIG. 4A is a table conceptually illustrating the operationidentification table stored in the memory 212 of the brain wave sensordevice 200 according to the first embodiment of the present disclosure.Specifically, the table stores feature points of brain wave signals ofthe user who performs operation inputs in a pseudo manner, sampled inadvance, and the table is used to determine that the user intends toperform what kind of pseudo operation. According to FIG. 4A, an item offeature point information and an item of operation content informationare stored in association with each item of operation input IDinformation. The brain waves of the user are sampled in advance when theuser performs various operation inputs to the display 111 functioning asthe pseudo input interface. The feature point information is informationon a characteristic peak of a pseudo operation identified on the basisof a waveform obtained from each of the sampled brain waves. Theoperation content information is information indicating the content of apseudo operation performed by the user when each item of the featurepoint information is obtained. In the present embodiment, for example, afeature point of a brain waveform detected by the brain wave sensor 211is compared with each feature point stored in the operationidentification table of FIG. 4A, and when the feature point of thedetected brain waveform matches feature point information “Ti”, a“touch” input is identified as pseudo operation content information.Then, from the brain wave sensor device 200, operation input IDinformation “O1” corresponding to the “touch” input is output asintention information to the terminal device 100.

FIG. 4B is a table conceptually illustrating the processingidentification table stored in the memory 113 of the terminal device 100according to the first embodiment of the present disclosure.Specifically, the table stores the operation input ID informationreceived as the intention information from the brain wave sensor device200, in association with the content of processing identified on thebasis of the operation input ID information. According to FIG. 4B, anitem of the operation input ID information and the content of processingare stored in association with each item of processing ID information.The operation input ID information is information in which a pseudooperation which the user intends to input is identified by the brainwave sensor device 200. The processing content information isinformation indicating the content of processing executed by theprocessor 112 on the basis of a predetermined pseudo operation havingbeen performed on the pseudo input interface by the user. In the presentembodiment, for example, when an item of the operation input IDinformation “O1” is received from the brain wave sensor device 200, theprocessor 112 refers to the processing identification table, determinesprocessing content information (“transition to next screen”) associatedwith an item of the operation input ID information “O1”, and executesprocessing (communication with the server device 300, drawingprocessing, etc.) to transition to the next screen.

3. Processing Sequence Between Devices

FIG. 5 is a diagram illustrating a processing sequence performed betweenthe terminal device 100, the brain wave sensor device 200, and a serverdevice 300 according to the first embodiment of the present disclosure.Specifically, FIG. 5 illustrates an example of an operation inputperformed when a game application “fighting game A” shown in FIG. 2 isactivated and a title screen is displayed, on the terminal device 100.FIG. 5 shows a sequence of interruption processing started by detectingan intention of the user to perform an operation input and transmittingintention information generated on the basis of the detection, by thebrain wave sensor device 200. FIG. 5 illustrates an example oftransmission and reception of information to and from the server device300 performed by operation input on the title screen of FIG. 2. This,however, merely shows an example. In other words, the user may be causedto perceive the situation through not only the title screen but alsoanother component. Furthermore, processing not using the server device300 may be performed or processing by using another server device oranother terminal device may be performed.

In FIG. 5, when the title screen is displayed on the display 111, theterminal device 100 sets the display 111 and the hardware key 116 as thepseudo input interface (S11). The user who perceives the pseudo inputinterface through the display 111 determines whether to touch thedisplay 111 or press the hardware key 116. Then, as a result of thedetermination, the user makes a decision to touch (operate) the display111 to cause a transition to a start screen. During making thisdecision, the processor 213 of the brain wave sensor device 200 monitorsfeature points of waveforms output from the brain wave sensor 211 havingthe electrodes El to En as needed (S12). Each time each of the featurepoints is obtained, the processor 213 refers to the operationidentification table illustrated in FIG. 4A and determines matching ofthe obtained feature point with a feature point of an operation inputperformed by the user, sampled in advance. When the feature points donot match, the processor 213 returns to monitoring a waveform again.

On the other hand, when the feature points match, the processor 213refers to operation content information associated with the featurepoint matching the obtained feature point and identifies the content ofthe operation input which the user intends to perform. In the presentembodiment, as described above, a decision made by the user to performan operation input to touch the touch sensor 115 matches the featurepoint of an item of operation input ID information “O1”. Therefore, theprocessor 213 generates an item of the intention information includingat least an item of the operation input ID information “O1” and performscontrol so as to transmit the intention information (T11) to theterminal device 100 via the external communication interface 214.

Next, the terminal device 100 refers to the processing identificationtable illustrated in FIG. 4B and starts processing associated in advancewith the operation input ID information received as the intentioninformation (S13). In the present embodiment, firstly, the terminaldevice 100 transmits user information (T12) stored in the memory 113 tothe server device 300. Then, the server device 300 having received theuser information authenticates the user (S14), and when the user isauthenticated as a valid user of the game application “fighting game A”,game data (T13) of the user stored in the server device 300 istransmitted to the terminal device 100.

The terminal device 100 sequentially receives the game data transmittedfrom the server device 300 (S15). When the reception of the game data iscompleted, the terminal device 100 starts drawing processing fordisplaying the start screen on the display 111 by using the receivedgame data or drawing information on the start screen stored in thememory 113 (S16).

Here, in the present embodiment, detection of actual touching orpressing the display 111 or hardware key 116 functioning as the pseudoinput interface is not required, in this process. That is, regardless ofthe presence or absence of the actual operation of the display 111 andhardware key 116, the terminal device 100 performs the processingaccording to S13 to S16, on the basis of the operation input IDinformation received as the intention information. In other words, evenif the user's actual operation on the display 111 or hardware key 116functioning as the pseudo input interface is finished after thedetection of the intention to perform the operation, the terminal device100 does not perform detection of the actual operation input (S111) ordoes not transmit operation information (T111) to the server device 300triggered with the actual operation input as a trigger.

4. Process Performed in Terminal Device 100

FIG. 6 is a flowchart illustrating a process performed in the terminaldevice 100 according to the first embodiment of the present disclosure.Specifically, the process relates to interruption processing started byreceiving intention information from the brain wave sensor device 200.

According to FIG. 6, the processor 112 displays the title screenillustrated in FIG. 2 on the display 111, sets the display 111 and thehardware key 116 as the pseudo input interface, and starts the process(S101). Then, the processor 112 determines whether operation input IDinformation is received as intention information from the brain wavesensor device 200 via the external communication interface 117 (S102).Then, when the intention information is received, on the basis of theoperation input ID information, pseudo operation which the user is aboutto perform is identified as touching the display 111. Then, processingassociated with the pseudo touch operation is performed on the titlescreen.

Specifically, the processor 112, firstly, controls the externalcommunication interface 117 to transmit user information stored in thememory 113 to the server device 300 (S103). Then, the processor 112controls the server device 300 to authenticate the user and startreception of game data necessary to execute the game application (S104).Next, the processor 112 starts drawing the next screen (start screen)sequentially on the basis of the received game data and drawinginformation stored in the memory 113 (S105).

Here, in the present embodiment, detection of actual touching orpressing the display 111 or hardware key 116 functioning as the pseudoinput interface is not required, in this process. That is, regardless ofthe presence or absence of the actual operations on the display 111 andhardware key 116, the processor 112 executes the processing according toS103 to S105, on the basis of the operation input ID informationreceived as the intention information. In other words, even if theuser's actual operation on the display 111 or hardware key 116functioning as the pseudo input interface is finished after thedetection of the intention to perform the operation, the terminal device100 does not perform detection of the actual operation input (S201) ordoes not perform transmission of operation information to the serverdevice 300 (S202) with the actual operation input as a trigger.

Note that when the intention of the user to perform the pseudo operationon the pseudo input interface is detected and the operation contentthereof is identified, feedback according to the identified operationmay be given to the user. Specifically, the memory 113 stores a feedbacktable in which feedback to be given is associated with each identifiedoperation input ID. Then, when the operation content is identified byreceiving the operation input ID information, the processor 112 refersto the table and determines the feedback according to the identifiedoperation. Examples of the feedback include vibration drive using avibration motor, lighting of an LED, display of an indicator on ascreen, and the like.

As described above, in the present embodiment, the brain wave sensordevice 200 detects an intention of the user to perform an operation, andthereby it is possible to cause the terminal device 100 to performcorresponding processing. Meanwhile, the terminal device 100 includesthe display 111 and the hardware key 116 which function as the pseudoinput interface, and although the execution of the processing itself isperformed on the basis of the intention detected by the brain wavesensor device 200, it is possible to give an operational feeling to theuser as if he/she actually operates the device 100.

SECOND EMBODIMENT

In the first embodiment, the entire display 111 and the hardware key 116which function as the pseudo input interface in the title screen havebeen described. That is, a description has been given of the brain wavesensor device 200 detecting the intention of the user to perform thepseudo operation on the display 111 or hardware key 116 to perform theprocessing on the basis of the detected intention. In a secondembodiment, a description will be given of a display 111, not the wholearea of which is caused to function as a pseudo input interface but apartial area of which is caused to function as the pseudo inputinterface. Note that the present embodiment is similar to the firstembodiment in configuration, processing, and procedure, except for thepoints described below specifically. Therefore, detailed descriptionthereof will be omitted.

FIG. 7 is a diagram conceptually illustrating processing in a systemaccording to the second embodiment of the present disclosure. Accordingto FIG. 7, one or a plurality of icons (icons 21 and 22) are displayedin at least partial areas of the display 111. In the present embodiment,the areas in which the icons 21 and 22 are displayed are caused tofunction as the pseudo input interface, and the other area in which, forexample, “fighting game A” is displayed is not caused to function as thepseudo input interface but to merely function as a display area.

Firstly, a brain wave sensor device 200 mounted to a user 11 detects anintention of the user to touch any of the icon 21 and the icon 22 whichfunction as the pseudo input interface with a pointer 12. Then, when theintention of the user is detected, processing associated therewith isperformed, according to the detected intention of the user. For example,when the intention of the user to touch (so-called short press) a starticon 21, is detected, the terminal device 100 performs the processingrelating to the display of a start screen identified on the basis ofinformation of the intention. At this time, the processing is performedon the basis of the detected intention of the user 11, and it is notnecessary to actually touch the icon 21 with the pointer 12 of the user11. In other words, regardless of whether the user operates the icons 21and 22, processing corresponding to the detected intention is performed.

FIG. 8A is a table conceptually illustrating an operation identificationtable stored in a memory 212 of the brain wave sensor device 200according to the second embodiment of the present disclosure.Specifically, the table stores feature points of brain wave signals ofthe user who performs operation inputs in a pseudo manner, sampled inadvance, and the table is used to determine that the user intends toperform what kind of pseudo operation. According to FIG. 8A, an item offeature point information and an item of operation content informationare stored in association with each item of operation input IDinformation. The brain wave of the user is sampled in advance when theuser intends to perform an operation to each of the icons functioning asthe pseudo input interface. The feature point information is informationon a characteristic peak identified on the basis of a waveform obtainedfrom the sampled brain wave. The operation content information isinformation indicating the content of a pseudo operation performed bythe user when each item of the feature point information is obtained andis used to identify what operation is performed on which icon. In thepresent embodiment, for example, a feature point of a brain waveformdetected by the brain wave sensor 211 is compared with each featurepoint stored in the operation identification table of FIG. 8A, and whenthe feature point of the detected brain waveform matches an item offeature point information “'S1”, “touch icon 21 (short press)” isidentified as pseudo operation content information. Then, from the brainwave sensor device 200, an item of operation input ID information “Q1”corresponding to “touch” input is output as the intention information tothe terminal device 100.

FIG. 8B is a table conceptually illustrating a processing identificationtable stored in a memory 113 of the terminal device 100 according to thesecond embodiment of the present disclosure. Specifically, the tablestores the operation input ID information received as the intentioninformation from the brain wave sensor device 200, in association withthe content of processing identified on the basis of the operation inputID information. According to FIG. 8B, an item of the operation input IDinformation and the content of processing are stored in association witheach item of processing ID information. The operation input IDinformation is information in which a pseudo operation which the userintends to input is identified by the brain wave sensor device 200. Theprocessing content information is information indicating the content ofprocessing executed by the processor 112 on the basis of a predeterminedpseudo operation having been performed on the pseudo input interface bythe user. In the present embodiment, for example, when the item of theoperation input ID information “Q1” is received from the brain wavesensor device 200, the processor 112 refers to the processingidentification table, determines processing content information(“display the start screen”) associated with the item of the operationinput ID information “Q1”, and executes processing (communication withthe server device 300, drawing processing, etc.) to display the startscreen.

In the present embodiment, a processing sequence performed between theterminal device 100, the brain wave sensor device 200, and the serverdevice 300 and a process performed by the processor 112 of the terminaldevice 100 are not illustrated. However, as in the first embodiment,predetermined processing is performed by detecting the intention of theuser to operate the icons 21 and 22 by the brain wave sensor device 200.Meanwhile, even if the user's actual operations on the icons 21 and 22functioning as the pseudo input interface are performed, the terminaldevice 100 does not perform processing to detect the actual operationinput nor use the actual operation inputs as a trigger for displayingthe start screen.

As described above, in the present embodiment, the brain wave sensordevice 200 detects an intention of the user to perform an operation, andthereby it is possible to cause the terminal device 100 to performcorresponding processing. Meanwhile, the terminal device 100 displaysthe icons 21 and 22 functioning as the pseudo input interface, andalthough the execution of the processing itself is performed on thebasis of an intention of the user detected by the brain wave sensordevice 200, it is possible to give an operational feeling to the user asif he/she actually operates the device 100.

THIRD EMBODIMENT

In the first embodiment, the entire display 111 and the hardware key 116which function as the pseudo input interface in the title screen havebeen described. In the second embodiment, at least partial areas of thedisplay, specifically, the icons 21 and 22 which function as the pseudoinput interface in the title screen have been described. In the thirdembodiment, a description is given of hardware keys 24 and 25 fixedlyprovided on a housing of a terminal device 100 and caused to function asthe pseudo input interface, instead of causing the display 111 or theicons 21 and 22 displayed on the display 111 to function as the pseudoinput interface. Note that the present embodiment is similar to thefirst embodiment in configuration, processing, and procedure, except forthe points described below specifically. Therefore, detailed descriptionthereof will be omitted.

FIG. 9 is a diagram conceptually illustrating processing in a systemaccording to a third embodiment of the present disclosure. According toFIG. 9, the terminal device 100 includes the hardware keys 24 includingdirection keys and an enter key, and the hardware keys 25 includingnumeric keys provided corresponding to the numerals 1 to 0, and thehardware keys 24 and 25 are fixedly provided on a surface of thehousing. In the present embodiment, these hardware keys 24 and 25function as a pseudo input interface. In the example of FIG. 9, adescription will be given of inputting characters (numbers and the like)by the user in a screen of a character input application displayed on adisplay 111. Specifically, the display 111 shows a cursor (next to theright side of “G”) indicating a position to which next character is tobe input, together with characters (“A” to “G”) having been input.

Firstly, a brain wave sensor device 200 mounted to a user 11 detects anintention of the user to press with a pointer 12 any of the hardwarekeys 24 and 25 functioning as the pseudo input interface. Then, when theintention of the user is detected, processing associated therewith isperformed, according to the detected intention of the user. For example,when the intention of the user to press a left arrow key of the hardwarekeys 24 is detected, the terminal device 100 performs processingidentified on the basis of information of the intention. At this time,the processing is performed on the basis of the detected intention ofthe user 11, and it is not necessary to actually press any of thehardware keys 24 with the pointer 12 of the user 11. In other words,regardless of whether the user operates any of the hardware keys 24 or25, processing corresponding to the detected intention is performed.

FIG. 10A is a table conceptually illustrating an operationidentification table stored in a memory 212 of the brain wave sensordevice 200 according to the third embodiment of the present disclosure.Specifically, the table stores feature points of brain wave signals ofthe user who performs operation inputs in a pseudo manner, sampled inadvance, and the table is used to determine that the user intends toperform what kind of pseudo operation. According to FIG. 10A, an item offeature point information and an item of operation content informationare stored in association with each item of operation input IDinformation. A brain wave of the user is sampled in advance when theuser intends to press any of the hardware keys 24 and 25 functioning asthe pseudo input interface. The feature point information is informationon a characteristic peak identified on the basis of a waveform obtainedfrom the sampled brain wave. The operation content information isinformation indicating the content of a pseudo operation performed bythe user when each item of the feature point information is obtained andis used to identify that the user intends to perform an operation onwhich hardware key. In the present embodiment, for example, a featurepoint of a brain waveform detected by the brain wave sensor 211 iscompared with each feature point stored in the operation identificationtable of FIG. 10A, and when the feature point of the detected brainwaveform matches an item of feature point information “U1”, “operationof up arrow key” is identified as pseudo operation content information.Then, from the brain wave sensor device 200, an item of operation inputID information “R1” corresponding to “operation of up arrow key” isoutput as the intention information to the terminal device 100.

FIG. 10B is a table conceptually illustrating a processingidentification table stored in a memory 113 of the terminal device 100according to the third embodiment of the present disclosure.Specifically, the table stores the operation input ID informationreceived as the intention information from the brain wave sensor device200, in association with the content of processing identified on thebasis of the operation input ID information. According to FIG. 10B, anitem of the operation input ID information and the content of processingare stored in association with each item of the processing IDinformation. The operation input ID information is information in whicha pseudo operation which the user intends to input is identified by thebrain wave sensor device 200. The processing content information isinformation indicating the content of processing executed by theprocessor 112 on the basis of a predetermined pseudo operation havingbeen performed on the pseudo input interface by the user. In the presentembodiment, for example, when the item of the operation input IDinformation “R1” is received from the brain wave sensor device 200, theprocessor 112 refers to the processing identification table, determinesprocessing content information (“upward movement of cursor”) associatedwith the item of the operation input ID information “R1”, and executesprocessing to move the cursor displayed on the display 111 upward.

In the present embodiment, a processing sequence performed between theterminal device 100, the brain wave sensor device 200, and the serverdevice 300 and a process performed by the processor 112 of the terminaldevice 100 are not illustrated. However, as in the first and secondembodiments, predetermined processing is performed by detecting anintention of the user to operate any of the hardware keys 24 and 25 bythe brain wave sensor device 200. Meanwhile, even if a user's actualoperation is performed on any of the hardware keys 24 and 25 functioningas the pseudo input interface, the terminal device 100 does not performprocessing to detect the actual operation input nor use the actualoperation input as a trigger for displaying a start screen.

As described above, in the present embodiment, the brain wave sensordevice 200 detects an intention of the user to perform an operation, andthereby it is possible to cause the terminal device 100 to performcorresponding processing. Meanwhile, the terminal device 100 includesthe hardware keys 24 and 25 functioning as the pseudo input interface,and the user can have an operational feeling as if he/she actuallyoperates the terminal device 100 although the processing itself isperformed on the basis of the intention detected by the brain wavesensor device 200. In particular, the hardware key enables the user todirectly feel the click feeling of the key, and higher operationalfeeling can be obtained.

FOURTH EMBODIMENT

In the first to third embodiments, a description has been given of thebrain wave sensor device 200 which detects a brain wave of a user,compares a brain wave of the user sampled in advance with the detectedbrain wave to identify information on an intention of the user will bedescribed. In the fourth embodiment, signals obtained by performingprocessing, such as A/D conversion, on brain waves of a user detected byelectrodes El to En are transmitted to a terminal device 100, andintention information on an intention of the user is identified in aprocessor 112 of the terminal device 100. Note that the presentembodiment is similar to the first to third embodiments inconfiguration, processing, and procedure, except for the pointsdescribed below specifically. Therefore, detailed description thereofwill be omitted.

In the present embodiment, as described above, to identify the intentioninformation on the intention of the user in the processor 112 of theterminal device 100, the operation identification tables illustrated inFIGS. 4A, 8A, and 10A are stored in the memory 113 of the terminaldevice 100. Then, between S12 and S13 shown in FIG. 5, the intentioninformation (T11) is received in the first to third embodiments, but abrain wave signal detected by the brain wave sensor 211 is received inthe present embodiment. Then, when receiving the brain wave signal, theprocessor 112 refers to the operation identification tables stored inthe memory 113 to identify the intention information on the intention ofthe user. Then, on the basis of the identified intention information onthe intention of the user, processing based on the intention informationon the intention of the user is performed, as in the first to thirdembodiments. The subsequent processing is the same as those in the firstto third embodiments.

As described above, in the present embodiment, the brain wave sensordevice 200 detects an intention of the user to perform an operation, andthereby it is possible to cause the terminal device 100 to performcorresponding processing. Meanwhile, the terminal device 100 includesthe hardware keys 24 and 25 functioning as the pseudo input interface,and the user can have an operational feeling as if he/she actuallyoperates the terminal device 100 although the processing itself isperformed on the basis of the intention detected by the brain wavesensor device 200.

Others

In the first to fourth embodiments, use of the brain wave sensor device200 has been described. However, in the present disclosure, theintention information on the intention of the user may only be detected,and it is possible to use various signals in addition to the brain wave.As an example, it is also possible to use a sensor device configured todetect an electric signal detected upon movement of a muscle to which anintention of the user is transmitted from a brain.

In addition, it is also possible to configure the system byappropriately combining or replacing component elements described in theabove embodiments.

The processing and procedures described herein may also be implementednot only by those explicitly stated for the embodiments but also bysoftware, hardware, or any combination thereof. Specifically, theprocessing and procedures described herein are achieved by implementinglogic corresponding to the processing, in a medium, such as anintegrated circuit, volatile memory, non-volatile memory, magnetic disk,or optical storage. Furthermore, the processing and procedures describedherein may be implemented as computer programs so that various computersincluding terminal devices and server devices are caused to execute thecomputer programs.

Even though the processing and procedures described herein are describedas being performed by a single device, software, component, or module,such processing and procedures may be performed by a plurality ofdevices, a plurality of software applications, a plurality ofcomponents, and/or a plurality of modules. Furthermore, even thoughvarious information is described herein as being stored in a singlememory or storage unit, such information may be distributed and storedin a plurality of memories included in a single device or in a pluralityof memories distributed and arranged in a plurality of devices.Furthermore, the software and hardware elements described herein may beachieved by being integrated with each other into a small number ofcomponent elements or by being divided into a large number of componentelements.

The terminal device, program, method, and system being thus described,it will be apparent that the same may be varied in many ways. Suchvariations are not to be regarded as a departure from the spirit andscope of the invention, and all such modifications as would be apparentto one of ordinary skill in the art are intended to be included withinthe scope of the following claims.

What is claimed is:
 1. A terminal device comprising: a pseudo input interface configured to make a user to perceive a pseudo operation; an external communication interface configured to receive intention information indicating an intention of the user from a sensor device, the sensor device being wiredly or wirelessly connected to the external communication interface, the sensor device detecting the intention of the user who intends to perform an intended operation with respect to the pseudo input interface; a memory configured to store computer readable instructions; and a processor configured to execute the computer readable instructions so as to: identify the intended operation by the user based on the intention information received by the external communication interface, regardless of an actual operation perceived by the user with respect to the pseudo input interface; and perform a processing corresponding to the identified intended operation.
 2. The terminal device according to claim 1, further comprising: a display configured to show an image, wherein the pseudo operation perceived by the user is an operation with respect to the display.
 3. The terminal device according to claim 1, further comprising: a display configured to show an image, wherein the pseudo input interface is an icon displayed on the display as the image so that when the icon is observed by the user, the pseudo operation is perceived by the user.
 4. The terminal device according to claim 1, wherein the pseudo input interface is an operation key fixedly provided on a housing of the terminal device.
 5. The terminal device according to claim 1, wherein the external communication interface transmits and receives predetermined information to and from a server device that is remotely installed, and the processing corresponds to the transmission and the reception of the predetermined information to and from the server device.
 6. The terminal device according to claim 1, wherein the processor executes the processing corresponding to the identified intended operation without activating the actual operation performed with respect to the pseudo input interface by the user, even when the pseudo input interface is actually operated by the user.
 7. The terminal device according to claim 1, wherein the sensor device includes one or a plurality of electrodes for detecting a brain wave of the user.
 8. The terminal device according to claim 1, wherein the processor is configured to provide feedback corresponding to the identified intended operation to the user.
 9. A system comprising: the terminal device according to claim 1; and a sensor device wiredly or wirelessly connected to the terminal device to detect the intention of the user perceiving one or a plurality of operation inputs with respect to the pseudo input interface of the terminal device.
 10. A non-transitory computer-readable storage medium storing a computer program product embodying computer readable instructions for causing a computer to execute a process by a processor so as to perform the steps of: making a user to perceive a pseudo operation via a pseudo input interface; receiving intention information indicating an intention of the user from a sensor device, the sensor device detecting the intention of the user who intends to perform an intended operation with respect to the pseudo input interface; identifying the intended operation by the user based on the intention information, regardless of an actual operation perceiving by the user with respect to the pseudo input interface; and performing a processing corresponding to the identified intended operation.
 11. A method for causing a processor in a computer to execute a process, the method comprising executing on the processor the steps of: making a user to perceive a pseudo operation via a pseudo input interface; receiving intention information indicating an intention of the user from a sensor device, the sensor device detecting the intention of the user who intends to perform an intended operation with respect to the pseudo input interface; identifying the intended operation by the user based on the intention information regardless of an actual operation perceiving by the user with respect to the pseudo input interface; and performing a processing corresponding to the identified intended operation.
 12. A terminal device comprising: a pseudo input interface configured to make a user to perceive a pseudo operation; an external communication interface configured to wiredly or wirelessly receive a signal indicating an intention of the user who intends to perform an intended operation with respect to the pseudo input interface; a memory configured to store computer readable instructions and a correspondence relationship between the signal and the intention of the user; and a processor configured to execute the computer readable instructions so as to: generate intention information indicating the intention of the user based on the received signal and the correspondence relationship stored in the memory, regardless of an actual operation perceived by the user with respect to the pseudo input interface; identify the intended operation by the user in response to the generated intention information; and execute a processing corresponding to the identified intended operation. 